[FLINK-40491][core] Extend BinaryVariant with TIME, TIMESTAMP_NS, TIMESTAMP_LTZ_NS primitives by manner · Pull Request #29050 · apache/flink

manner · 2026-08-31T13:04:58Z

What is the purpose of the change

The variant binary encoding spec (see Variant Encoding) defines primitive type codes 17-20 for TIME, TIMESTAMP_LTZ_NS, TIMESTAMP_NS, and UUID that are currently missing in Flink.
This PR adds support for codes 17-19 (TIME, TIMESTAMP_LTZ_NS, TIMESTAMP_NS) to BinaryVariant, so that a LocalTime and a nanosecond-precision Instant/LocalDateTime can be represented in a Variant without lossy truncation to microseconds.
UUID (code 20) is intentionally out of scope here and will be added in a separate ticket.

Brief change log

Added BinaryVariantUtil constants for primitive codes 17-19, extended getType()/valueSize()/getLong() to handle them, and added a TIME_FORMATTER for JSON rendering
Added Variant.Type.TIME/TIMESTAMP_NS/TIMESTAMP_LTZ_NS and the corresponding getTime()/getDateTimeNanos()/getInstantNanos() accessors to the Variant interface
Added VariantBuilder.of(LocalTime); made of(Instant)/of(LocalDateTime) precision-aware so a value with no sub-microsecond component keeps using the existing compact micros encoding, and only switches to the new nanosecond encoding when the value actually needs it
Implemented appendTime/appendTimestampNanos/appendTimestampLtzNanos in BinaryVariantInternalBuilder, and the matching read/get()/toJson() support in BinaryVariant

Verifying this change

This change added tests and can be verified as follows:

Round-trip and generic get() dispatch tests for the new types (BinaryVariantTest#testScalarVariant)
Dedicated precision-dispatch tests verifying Instant/LocalDateTime pick the existing TIMESTAMP_LTZ/TIMESTAMP encoding for microsecond-aligned values and the new TIMESTAMP_LTZ_NS/TIMESTAMP_NS encoding otherwise, including that the mismatched accessor throws VariantTypeException (BinaryVariantTest#testNanosecondPrecisionVariant)
A test verifying that LocalTime silently truncates below microsecond precision, since TIME has no nanosecond-precision counterpart in the variant spec (BinaryVariantTest#testTimeSubMicrosecondTruncation)
JSON rendering assertions for TIME/TIMESTAMP_NS/TIMESTAMP_LTZ_NS (BinaryVariantTest#testToJsonScalar)

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes (Variant and VariantBuilder are @PublicEvolving; this adds new enum constants and new interface methods)
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? not documented (documentation is tracked separately in FLINK-40494)

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Claude Sonnet 5 (Claude Code)

…ESTAMP_LTZ_NS primitives

flinkbot · 2026-08-31T13:10:21Z

CI report:

c7dc74a Azure: PENDING
5eea899 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

raminqaf

Thanks for the PR @manner! Left two comments

raminqaf · 2026-08-31T15:20:13Z

+        assertThat(dateTimeVariant.getType()).isEqualTo(Variant.Type.TIMESTAMP_NS);
+        assertThat(dateTimeVariant.getDateTimeNanos()).isEqualTo(nanoLocalDateTime);
+        assertThat(dateTimeVariant.get()).isEqualTo(nanoLocalDateTime);
+        assertThatThrownBy(dateTimeVariant::getDateTime).isInstanceOf(VariantTypeException.class);


Try this case

LocalDateTime nanoLocalDateTime2 = LocalDateTime.of(2300, 1, 1, 0, 0, 0, 1); Variant dateTimeVariant2 = builder.of(nanoLocalDateTime2);

This will overflow. We should handle the error better

Good catch! I'll add proper handling

Actually the same error will currently happen when using a microsecond timestamp such as
LocalDateTime nanoLocalDateTime = LocalDateTime.of(2300, 1, 1, 0, 0, 0, 0);
Instant.until()s function for calculating microseconds between two instants just uses the function for calculating nanoseconds between two instants and then divides the result by 1000. This then also results in an overflow.

So this was a pre-existing bug, that only surfaced now.

raminqaf

LGTM @manner! Thanks for the contribution! 🚀

snuyanzin · 2026-09-01T18:06:11Z

after looking one more time the thing I didn't get
why don't we update VariantCastUtils with newly added types?

I guess we should, or?

flink/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java

Lines 294 to 312 in 878844f

    
           case TIMESTAMP: 
        
               // A wall-clock value needs no zone shift, which is what UTC_ZONE achieves here. A 
        
               // variant keeps microseconds, so the precision is always 6. 
        
               value = 
        
                       DateTimeUtils.formatTimestamp( 
        
                               TimestampData.fromLocalDateTime(variant.getDateTime()), 
        
                               DateTimeUtils.UTC_ZONE, 
        
                               TIMESTAMP_PRECISION); 
        
               break; 
        
           case TIMESTAMP_LTZ: 
        
               value = 
        
                       DateTimeUtils.formatTimestamp( 
        
                               TimestampData.fromInstant(variant.getInstant()), 
        
                               sessionZone, 
        
                               TIMESTAMP_PRECISION); 
        
               break; 
        
           case NULL: 
        
               // Only reachable for a NOT NULL target. A nullable target maps a null-valued 
        
               // variant to SQL NULL before this method is called.

raminqaf · 2026-09-02T05:56:02Z

after looking one more time the thing I didn't get why don't we update VariantCastUtils with newly added types?

I guess we should, or?

flink/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java

Lines 294 to 312 in 878844f

case TIMESTAMP:

// A wall-clock value needs no zone shift, which is what UTC_ZONE achieves here. A

// variant keeps microseconds, so the precision is always 6.

value =

DateTimeUtils.formatTimestamp(

TimestampData.fromLocalDateTime(variant.getDateTime()),

DateTimeUtils.UTC_ZONE,

TIMESTAMP_PRECISION);

break;

case TIMESTAMP_LTZ:

value =

DateTimeUtils.formatTimestamp(

TimestampData.fromInstant(variant.getInstant()),

sessionZone,

TIMESTAMP_PRECISION);

break;

case NULL:

// Only reachable for a NOT NULL target. A nullable target maps a null-valued

// variant to SQL NULL before this method is called.

Good point. We are planing to deliver the casting changes and support in a follow-up ticket/PR https://issues.apache.org/jira/browse/FLINK-40492

[FLINK-40491][core] Extend BinaryVariant with TIME, TIMESTAMP_NS, TIM…

5b4a097

…ESTAMP_LTZ_NS primitives

[FLINK-40491][core] Fix formatting

c7dc74a

raminqaf reviewed Aug 31, 2026

View reviewed changes

manner added 2 commits August 31, 2026 17:47

[FLINK-40491][core] Add comment about truncating LocalTime

ffd3b20

[FLINK-40491][core] Reject out-of-range nanosecond-precision timestamps

058498a

manner requested a review from raminqaf August 31, 2026 15:53

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Aug 31, 2026

snuyanzin reviewed Aug 31, 2026

View reviewed changes

Comment thread flink-core/src/main/java/org/apache/flink/types/variant/BinaryVariantUtil.java

snuyanzin reviewed Aug 31, 2026

View reviewed changes

Comment thread flink-core/src/main/java/org/apache/flink/types/variant/BinaryVariantUtil.java

raminqaf reviewed Sep 1, 2026

View reviewed changes

Comment thread flink-core/src/main/java/org/apache/flink/types/variant/BinaryVariantUtil.java Outdated

raminqaf reviewed Sep 1, 2026

View reviewed changes

Comment thread flink-core/src/test/java/org/apache/flink/types/variant/BinaryVariantTest.java Outdated

manner added 2 commits September 1, 2026 12:06

[FLINK-40491][core] Add nano precision to Time to check truncation

19c6ec8

[FLINK-40491][core] Refactor comments to Javadoc and fix inconsistencies

5eea899

raminqaf approved these changes Sep 1, 2026

View reviewed changes

Navigation Menu

Uh oh!

[FLINK-40491][core] Extend BinaryVariant with TIME, TIMESTAMP_NS, TIMESTAMP_LTZ_NS primitives - #29050

[FLINK-40491][core] Extend BinaryVariant with TIME, TIMESTAMP_NS, TIMESTAMP_LTZ_NS primitives#29050
manner wants to merge 6 commits into
apache:masterfrom
manner:FLINK-40491

manner commented Aug 31, 2026

Uh oh!

flinkbot commented Aug 31, 2026 •

edited

Loading

Uh oh!

raminqaf left a comment

Uh oh!

raminqaf Aug 31, 2026

Uh oh!

manner Aug 31, 2026

Uh oh!

manner Sep 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raminqaf left a comment

Uh oh!

snuyanzin commented Sep 1, 2026 •

edited

Loading

Uh oh!

raminqaf commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

manner commented Aug 31, 2026

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

flinkbot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

raminqaf left a comment

Choose a reason for hiding this comment

Uh oh!

raminqaf Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

manner Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

manner Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raminqaf left a comment

Choose a reason for hiding this comment

Uh oh!

snuyanzin commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raminqaf commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flinkbot commented Aug 31, 2026 •

edited

Loading

snuyanzin commented Sep 1, 2026 •

edited

Loading